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ABSTRACT 

In evaluating this pilot project, which developed a 
computer system for assessing student progress and cost effectiveness 
as related to achievement of performance objectives, interviews were 
conducted with project participants, including project staff, school 
administrators, and the auto shop instructors* Project documents were 
reviewed and a brief questionnaire was used to obtain student 
opinions about their experiences with this man-machine system. The 
system will serve as a valuable change agent if the computer data is 
used to generate reports which can be acted upon by appropriate 
personnel. These reports are useful for pinpointing individual and 
class differences. Suggestions for future applications follow summary 
statements about the program's success in achieving its objectives. 
This document is related to a technical manual and a nontechnical 
manual, available in this issue as VT 018 581 and VT 018 580, 
respectively. (AG) 
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PBAPOL Project Evaluation • 
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\ * 
INTRODUCTION 

: 
s 

The PEAPOL Project objectives , as stated in performance terms in the 
project application, have been accomplished, with only a few modifications 
in data input. Performance objectives were written, a system was designed 
to accept inputs and generate the outputs specified, computer programs were 
written and tested, orientation sessions were held, PPBS-type budgets were 
organized for the autoshop classes, and pilot tests were conducted in the 
schools. The final objective, documentation, is nearly complete. 

In evaluating this effort to develop an instructional assessment sys- 
tem, interviews were conducted with project participants, including project 
staff, school administrators, and the autoshop class instructors. Project 
documents were studied. A brief questionnaire was used to obtain student 
opinions about their experiences with the prototype system. 

The purpose of these activities was to gather facts and opinions that 
would indicate: 

© How each project objective had been carried out. 

© Reactions o^ users to the instructional assessment system: their 
views of- its strengths, weaknesses, and most promising outcomes. 

© Effects of the system on users, and of users on the system. 

Information gathered about each objective is summarized and discussed 
briefly below. Objective 6, the pilot tests, "is discussed in greater detail, 
with remarks about aspects of the system which appeared crucial in this 
phase. Some additional observations are followed by suggestions for future 
applications and a statement summarizing this evaluation report. 

REVIEW OF PROJECT OBJECTIVES 

2. Performance Objectives. The performance oh-} actives written by the two 
instructors for their autoshop classes outlined tfisks to be accomplished for 
each objective in a few sentences. References wc-.ro giver, text materials 



students could use for completing each objective** After this pilot use, 
one instructor felt that his objectives would need about 25 percent revi«;ioni 
the other, 100 percent. Each had, in mind a form of basic overhaul in the 
presentation of his objectives* One recognized that he had used poor judg- 
ment in sequencing objectives (students aopeared to v/ant to take them in 
order, which was not always practical). The other instructor hoped to de- 
velop objectives in workbook style, where students filled in their answers. 
This, he felt, would provide an element of quality control that was missing 
with the present system. Both instructors thought that computer data about 
hours used and cost for each objective would be useful in evaluating and 
revising their performance objectives, as well as their overall course 
content. 

2. System Design. The types of information input into the system and the 
output generated by it were as described in the project proposal, except 
that lateness data and descriptive data pertaining to teachers were omitted. 
Perhaps the greatest weakness in the prototype system design resulted from 
the attempt to write performance objectives of equal difficulty and from 
allocating an equal cost per unit of time. As proposed by project staff, 
assigning multipliers to weight objectives for differences in cost and/or 
difficulty will produce more accurate data for program evaluation. The five- 
week lag designed into the class progress indicator may be too lont^; in the 
final printout two classes were still reported as "progressing normally," 
even though no time had been recorded for them for several weeks. 

3. Programming. Computer programs were written with consideration for the 
experimental nature of the project. Decisions to use a con<jcol program, to 
limit core size, and to provide special error -hand ling routines, which added 
to costs, appear to have been well made, since there were no instances re- 
ported where the computer components of the system seriously impaired its 
operation. Project staff suggested modif ications in the prototype which 
would significantly lower costs in a production model, e.g., using mark sense 
cards instead of key punching and using a larger core or disk storage. 



Of the 25 students responding to the questionnaire, 17 said they liked 
having their assignments stated as; performance objectives. 
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4. Orientation. Orientation for instructors, administrators, and students 
was minimal, as was admitted by the project director. Although satisfied at 
the time that questions had been answered, participants in the pilot project 
could not anticipate many questions and problems that would arise, with the 
pilot experience behind them, project staff will be better equipped to demon- 
strate to future participants how to use the system effectively. Orientation 
should be strengthened in all areas listed in the project proposal: syst em 
rationale, performance objective rationale, input requirements, and particu- 
larly, how to interpret and utilize information generated by the system. 

5. PPBS-Typo Budgets. These were developed for the three autoshop classes 
and incorporated into the system. Budget information was printed out weekly 
to provide a current status report on uses of project funds. Also, compari- 
sons of budgeted costs and actual expenditures per credited hour and per 
pupil could be made. In the printout for the final week, all three classes 
were slightly under the total dollars budgeted. 

6. Pilot Tests. The pilot tests in. the two high schools brought the FSAPOL 
system concept and design face-to-face with teachers and students in the 
classroom. The two school contexts in which the tests were conducted 

• were quite different: differences in school programming, in the instructors' 
teaching methods, and in their uses of the system led to somewhat different 
outcomes from the pilot tests at. the two schools. These differences wore 
apparent in data generated by the computer as well as in opinions expressed 
by teachers and students. 

For example, a striking difference is reported on the Student Suaaary 
Report for the last week of the three classes. The printouts show that the 
average number of performance objectives completed by each student at o-.,o 
school was nearly three times the number completed by students at the other 
school, yet the average number of objectives per hour was about the B*a a . 
These data reflected differences in course organization and class schooling, 
as well as teacher and stud-.nt attitudes toward using the system. 

During the pilot test.., it became apparent that how the assessment 
system output-was used affected its impact on instruction. For example, the 
most immediately useful output of the system appeared to be the Student 
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Sununary Reports This weekly roporl; was intended for posting for examina- 
tion by students, so that each could chock his progress • At one school , 
where the report was posted regularly throughout the semester, the instruc- 
tor said that his students were eager to receive the printouts each week. 
Of 15 of his students who responded to the question, "Have the weekly re- 
ports helped you in any way?" 14 said, "Yes*" The instri^ctor cited how one 
of the poorest students had "pulled himself up by his bootstraps , because 
of the readout." Most students said they had been helped by the report 
because it showed them how they were doing in class . At the other school 
reports were not posted regularly; seven out of u en students there said the 
reports had not helped them. 

The instructors had used (and planned to use further) output data for 
curriculum revision. Data on the status of individual students and on indi- 
vidual performance objectives could be compared at a point in time or fol- 
lowed over a period of time to note 'remarkable differences or changes as 
they occurred. Both instructors cited examples of how the printout data had 
brought to their attention needed changes in course content or teaching 
techniques. They said that while they could maintain similar records (e.g., 
of student time per objective) , processing these by hand would be very time- 
consuming and would produce information in less detail and less often. One 
instructor said he was developing forms similar to those used for the PEAPOL 
project. If deprived of the system, an aide could help him keep track of 
student time and performance. 

In the pilot tests the frequency of the computer-generated reports 
appeared to be an important asset, especially in the classroom. Both in- 
structors recommended that the Student Summary Report continue to be issued 
weekly (although other parts of the program might be printed out less often) . 
They agreed with students that knowledge of where they stood at all times on 
each objective was desirable. 

The time accounting capability of the system was a~major strength, par- 
ticipants seemed to think. One administrator said he felt that the major 
benefits of the system were not related to its cost accounting output, but 
to its effect on students 1 attitude toward time. In his view, tho system 
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provides a graphic way of making students aware of time so that they are 
more apt to make productive use of it. Both instructors supported this 
view; both planned to continue using the time clock in future classes. 
Students were not as enthusiastic , but 19 of the 25 responding to the ques- 
tionnaire said they didn*t mind punching the clock. , * 

Occasional "mistakes" in the pilot test printout reports were appar- 
ently caused by omissions or errors in data input.- Students were dis- 
gruntled when the computer failed to credit them with time on objectives or 
gave them unfair warning flags; this was usually due to a human failure, re- 
sulting from data not being input as required. As discussed earlier, there 
is need to refine the system programs to allow for weighting performance 
objectives for different levels of difficulty and cost. Data from the pilot 
project printouts will be useful in making these modifications, which should 
also correct discrepancies in issuing warning and merit flags. 

The element of quality control was a concern of many who participated 
in the pilot tests. All felt that this must continue to rest with the in- 
structor. The computer might help, however, by processing and reporting 
data on levels of performance as judged by the instructor. One instructor 
recommended adding a quality control component to the time and cost input/ 
output capabilities of the system and advocated including students' grades 
on the weekly Student Summary Report. 

Accurate cost information related to teaching each objective promised 
to be useful in budgeting for and justifying equipment expenditures and in 
scheduling equipment use. Cost data would al.so be useful for program evalu- 
ation in cost/benefit analyses of alternative teaching techniques and units 
of instruction. 

7. Documentation. Two of the project documents were reviewo'd for this 
evaluation; the "User's Manual" and the "Technical Manual." Reviews were by ■ 
an instructor, an administrator, and the data processing manager, all of 
whom agreed that the documents appeared to present information about the " 
prototype system clearly and completely. A summary statement briefly out- 
lining the -system in. lay terms and a step-by-step summary wore suggested 
by reviewers for inclusion in the "User's Manual" (apparently the summary 



ERIC 



-5- 



I 



provided was considered too technical) . Since neither of these documents 
was field-tested, it is to be expected that there will be place's where addi- 
tional information will be needed to clarify procedures. 

. • ' ! 

i 

ADDITIONAL OBSERVATIONS 

■ i 

! 

It has been suggested that the system could be used by administrators 
as a tool in personnel evaluation. While student performance data generated 
could be used as an element in instructor evaluation, these data should be 
related to individual differences in the way the teacher is using the system 
and to other circumstances, such as class scheduling. 

Costs of operation of a production version of the PEAPOL system ($.40 
■per week per class, as projected by project staff) are certainly within the 
budget of schools to which the system might be mado available. Instructor 
time must be added to this: the amount of instructor time reported for pre- 
paring inputs for the pilot system was 30 - 90 minutes per week. (Instruc- 
tors said the system had cut down on their paperwork,.) 

In writing their performance objectives, the two instructors had the 
opportunity to re-examine their course content and to reorganize and update 
its subject matter. The PEAPOL project provided thoin -with time to do this. 
It seems likely that, use of such a system (even temporary use) would have an 
impact on instruction by requiring teachers to perform this type of review 
and by providing feedback on the expected outcomes of their course revisions. 

SUGGESTIONS FOR OTHER APPLICATIONS 
IN EDUCATION 

While it seems particularly suited to skil} tr.-iining-type course*, 
imaginative teachers in nearly any subject area could use the PEAPOL system 
for a portion of their courses: -for example, as a motivational tool to get 
students through the more technical or mechanical aspects of academic courses 
Why should the entire course necessarily be monitor^!? (In e ff oct , this 
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appears to be what happened at one high school in the pilot study; only a 

i . 

portion of student tine in class was monitored •) ! j 

i 

The system might be developed as a program assessment tool for educa- 
tional research to yield data under more controlled circumstances that would 
help test variables in curriculum, teaching methods , scheduling , teacher- 
student relationships, etc. ' 

The system might be used for inservice and pre-service teacher educa- 
tion. The impact of. the pilot test on the instruction of the two teachers 
suggests that other teachers, particularly less experienced teachers, might 

benefit from experiences with the instructional assessment system. 

» 

SUMMARY STATEMENT 

By its designation, the PEAPOL project's purpose was to investigate the 
concept of program evaluation at the performance objective level* The pilot 
project demonstrated that computer monitoring of time required by students 
to complete individual performance objectives is useful for pinpointing dif- 
ferences in individual student performance over time or over a series of 
objectives; when these differences are detected by the instructor, he can 
attempt to discover the cause and find the needed remedy. In the same way 
differences in cost can be analyzed and used by instructors and administra- 
tors in adjusting program elements,, such as content, scheduling, and equip- 
ment purchases and use. The printout reports also document differences 
between classes. These differences, traced to their causes, may shed light 
for program evaluation by both instructors and administrators at a more 
comprehensive level than the performance objective level. 

The system can s6rve as a tool in diverse circumstances: teachers 
need not fear that, to use the system, they must fit their methods of in- 
struction to a strict format. However, the system will be a useful and 
beneficial change agent only if it is used; that is, if the required data 
is conscientiously input and if the generated reports are made available 
to, studied by, and acted upon by the appropriate participants. 
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Refinements in the prototype PEAPOL system, proposed by the project 
staff, will be necessary before this tool can operate more efficiently to 
provide accurate data for instructional assessment. It is hoped tffat the 
f effort to make these refinements will be supported, and that a production 
version of the tool will be developed for use in many other vocational edu- 
cation settings and elsewhere, 
| 
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